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ABSTRACT 


The  Thinking,  Risk,  and  Intelligence  Group  (TRIG),  one  of  three  groups  of  the  Adversarial  Intent 
Section  at  Defence  Research  and  Development  Canada  (DRDC)  Toronto,  is  unique  in  Canada 
given  its  objective  “to  support  Canada’s  defence  and  security  community  through  applied 
behavioural  science  aimed  at  promoting  human  effectiveness  in  risk  management  and 
intelligence  production”.  The  Group  Leader  of  TRIG,  Dr.  David  Mandel,  is  leading  an  Applied 
Research  Project  (ARP)  entitled  “Understanding  and  augmenting  human  capabilities  for 
intelligence  production”.  Under  the  auspices  of  this  ARP,  the  TRIG  at  DRDC  Toronto  held  a 
discovery  workshop  on  5  February,  2010  to  explore  and  discuss  the  utility  of  concept  map 
knowledge  modelling  within  the  Canadian  intelligence  analysis  community.  Several  government 
organizations  participated  in  a  half-day  workshop  that  included  two  sets  of  interactive,  group- 
oriented,  break-out  sessions  in  which  teams  engaged  in  discussion  activities  to  address  several 
objectives  related  to  a  concept  map  knowledge  model  and  concept  mapping  (CMapping)  in 
general.  Participants  derived  several  conclusions  relating  to  the  organization,  design  and  the 
potential  application  of  CMapping,  in  addition  to  several  implementation  issues  and  suggestions 
for  the  improvements.  This  report  summarises  the  impressions  of  participants  pertaining  to  the 
concept  map  tool  and  process,  and  key  points  and  outcomes  from  the  workshop. 
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EXECUTIVE  SUMMARY 


The  Thinking,  Risk,  and  Intelligence  Group  (TRIG),  one  of  three  groups  of  the  Adversarial  Intent 
Section  at  Defence  Research  and  Development  Canada  (DRDC)  Toronto,  is  unique  in  Canada 
given  its  objective  “To  support  Canada’s  defence  and  security  community  through  applied 
behavioural  science  aimed  at  promoting  human  effectiveness  in  risk  management  and 
intelligence  production”.  The  Group  Leader  of  TRIG,  Dr.  David  Mandel,  is  leading  an  Applied 
Research  Project  (ARP)  entitled  “Understanding  and  augmenting  human  capabilities  for 
intelligence  production”.  Under  the  auspices  of  this  ARP,  the  TRIG  at  DRDC  Toronto  held  a 
discovery  workshop  on  5  February,  2010  to  explore  and  discuss  the  utility  of  concept  map 
(CMap)  knowledge  modelling  within  the  Canadian  intelligence  analysis  community. 

Several  government  organizations  participated  in  a  half-day  workshop  that  included  two  sets  of 
interactive,  group-oriented,  break-out  sessions  in  which  teams  engaged  in  discussion  activities 
to  address  several  objectives  related  to  a  CMap  knowledge  model  and  concept  mapping 
(CMapping)  in  general.  The  workshop’s  format  balanced  formal  activities  with  open  discussion- 
based  interaction  and  drew  on  intelligence  analysis  expertise.  The  workshop  produced  thought- 
provoking  and  insightful  observations  on  factors  involved  in  CMapping  for  the  intelligence 
community  in  Canada. 

Two  broad  conclusions  were  shared  by  the  participants  at  this  discovery  workshop.  Firstly, 
CMapping  has  potential  utility  for  organizing  analytic  thought  and  structuring  the  thinking 
process,  and  it  is  useful  as  an  imaginative  or  visual  tool  to  “see  the  bigger  picture”  and  explore  a 
topic  in  depth.  As  a  visual  representation  of  an  analysis,  a  CMap  can  be  used  to  validate 
arguments  by  identifying  any  gaps  in  the  argument  and  to  communicate  ideas  and  foster 
discussion  about  particular  topics.  Secondly,  CMapping  was  determined  to  be  a  difficult 
process  that  takes  time  to  learn  and  may  not  be  amenable  to  quick  tactical  problem  analysis.  In 
addition,  while  there  are  many  benefits  of  using  CMaps  and  executing  a  CMapping  process,  this 
may  be  of  primary  benefit  to  people  who  are  visual  thinkers.  CMapping  may  therefore  not  be  a 
practical  process  to  implement.  Participants  derived  several  conclusions  relating  to  the 
organization,  design  and  the  potential  application  of  CMapping,  in  addition  to  several 
implementation  issues  and  suggestions  for  improvements. 

This  report  summarises  the  impressions  of  participants  pertaining  to  the  CMap  tool  and  process, 
and  key  points  and  outcomes  from  the  workshop.  The  third  section,  Summary  of  workshop 
break-out  group  sessions,  describes  the  process  and  outcomes  of  the  interactive  break-out 
sessions,  including  the  potential  uses,  benefits  and  drawbacks  of  CMapping.  The  fourth 
section,  Discussion  of  workshop  results,  provides  a  more  detailed  discussion  of  topics  related  to 
CMapping  including  the  potential  application  of  CMapping,  implementation  issues  of  CMapping 
and  suggestions  for  improvements. 
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1  INTRODUCTION 

This  report,  completed  under  contract  for  the  Thinking,  Risk,  and  Intelligence  Group  (TRIG),  one 
of  three  groups  of  the  Adversarial  Intent  Section  at  Defence  Research  and  Development 
Canada  (DRDC)  Toronto,  is  documentation  of  the  proceedings  of  a  workshop  entitled,  “Concept 
Map  Knowledge  Model  Workshop”.  Specifically,  this  report  documents  the  feedback  from  the 
Subject  Matter  Experts  (SMEs)  in  the  intelligence  community  and  the  outcomes  arising  from 
break-out  group  sessions  during  the  validation  workshop.  This  report  will  help  guide  the 
potential  application  of  concept  mapping  (CMapping)  to  intelligence  activities,  including  analysis, 
training,  communication,  and  resource  planning. 


1.1  Background 

The  Group  Leader  of  TRIG,  Dr.  David  Mandel,  is  leading  an  Applied  Research  Project  (ARP) 
entitled  “Understanding  and  augmenting  human  capabilities  for  intelligence  production”.  This 
ARP  is  conducted  for  Captain  (N)  Barber,  Director  of  Intelligence  Capabilities  at  Chief  of 
Defence  Intelligence  (CDI)  and  is  funded  through  the  Intelligence  Thrust  of  the  Command, 
Control,  Communications,  Computers,  Intelligence,  Surveillance  and  Reconnaissance  (C4ISR) 
Partner  Group.  The  objective  of  TRIG  is  to  support  Canada’s  defence  and  security  community 
through  applied  behavioural  science  aimed  at  promoting  human  effectiveness  in  risk 
management  and  intelligence  production. 

Under  the  auspices  of  this  ARP,  Dr.  Mandel  and  Dr.  Natalia  Derbentseva,  also  from  TRIG,  have 
developed  a  concept  map  (CMap)  knowledge  model1  of  intelligence  analysis  (hereafter 
CMKMIA).  CMapping  is  a  knowledge-visualization  and  modeling  technique  which  has  been 
used  by  DRDC  Toronto  to  visualize  the  domain  of  intelligence  analysis.  The  CMKMIA,  created 
in  CmapTools  (Novak  &  Cafias,  2008),  consists  of  a  collection  of  interlinked  CMaps 
supplemented  with  links  to  academic  literature,  technical  reports,  and  images.  The  interlinked 
CMaps  address  many  different  facets  of  the  intelligence  analysis  process,  including  products, 
challenges,  people,  organizations,  cognitive  processes,  training,  and  purposes.  The  purpose  of 
CMKMIA  was  to  demonstrate  the  potential  applications,  both  as  process  and  product,  of 
CMapping  to  an  audience  with  limited  familiarity  with  the  specific  approach,  and  to  generate 
their  feedback  on  the  tool  and  on  CMKMIA  itself. 


1.2  CMapping  Overview 

CMaps  are  “graphical  tools  for  organizing  and  representing  knowledge.  They  include  concepts, 
usually  enclosed  in  circles  or  boxes  of  some  type,  and  relationships  between  concepts  indicated 
by  a  connecting  line  linking  two  concepts.  Words  on  the  line,  referred  to  as  linking  words  or 
linking  phrases,  specify  the  relationship  between  the  two  concepts.  Figure  1-1  shows  an 


A  Concept  Map  Knowledge  Model  is  distinct  from  a  Concept  Map.  A  Concept  Map  is  a  single  map,  probably  presented  in  one  view, 
while  a  Knowledge  Model  is  a  set  of  interlinked  maps.  Both  may  contain  linked  document,  etc.  references. 
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example  of  a  concept  map  that  describes  the  structure  of  concept  maps  and  illustrates  the 
above  characteristics”  (Novak  &  Canas,  2008,  p.  1). 


Figure  1-1 :  A  CMap  Showing  the  Key  Features  of  CMaps  (Novak  &  Canas,  2008) 

Main  characteristics  of  CMaps  are: 

•  Concepts.  Novak  &  Canas  (2008,  p.1)  define  a  “concept  as  a  perceived  regularity  in  events 
or  objects,  or  records  of  events  or  objects,  designated  by  a  label”.  A  word  or  several  words 
are  used  as  a  label  for  most  concepts  although  symbols  such  as  +  or  %  are  also  sometimes 
used. 

•  Propositions.  Propositions  are  defined  by  Novak  &  Canas  (2008,  p.  1)  as  “statements  about 
some  object  or  event  in  the  universe,  either  naturally  occurring  or  constructed.  Propositions 
contain  two  or  more  concepts  connected  using  linking  words  or  phrases  to  form  a 
meaningful  statement. 

•  Hierarchical  arrangement.  “Concepts  are  represented  in  a  hierarchical  fashion  with  the  most 
inclusive,  most  general  concepts  at  the  top  of  the  map  and  the  more  specific,  less  general 
concepts  arranged  hierarchically  below.  Concept  maps  tend  to  be  read  progressing  from 
the  top  downward”  (Novak  &  Canas,  2008,  p.  1-2). 

•  Cross-links.  “These  are  relationships  or  links  between  concepts  in  different  segments  or 
domains  of  the  concept  map”  (Novak  &  Canas,  2008,  p.  2). 
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CMKMIA  was  developed  based  on  the  CMapping  methodology  described  in  Novak  and  Canas 
(2008).  The  DRDC  model  is  briefly  described  below. 


1.3  CMap  Knowledge  Model  of  Intelligence  Analysis  (CMKMIA) 

CMKMIA  was  developed  as  a  comprehensive  resource  on  the  topic  of  intelligence  analysis.  It 
depicts  interdependencies  among  various  issues  involved  in  intelligence  analysis  and  also 
serves  as  a  “knowledge  hub,”  which  provides  easy  access  to  attached  academic  and 
practitioner  literature  on  specific  topics.  The  model,  created  in  CmapTools,  consists  of  a 
collection  of  interlinked  CMaps  supplemented  with  links  to  academic  literature,  technical  reports, 
and  images.  See  Figure  1-2  for  the  top  level  map  of  the  CMKMIA. 


Focus  Question:  What  is  intelligence  analysis?  | 


Figure  1-2:  Top  level  map  of  CMKMIA 


1.4  CmapTools  Software  Toolkit 

The  CmapTools  software  toolkit  was  used  to  create  CMKMIA.  The  CmapTools  software  was 
developed  at  the  Institute  for  Human  and  Machine  Cognition.  It  has  been  made  available  for 
free  to  promote  the  sharing  of  knowledge.  The  software  enables  users  to  construct  and  modify 
CMaps  much  as  one  might  use  a  word  processor,  and  it  enables  users  to  collaborate  at  a 
distance  in  the  construction  of  their  maps,  and  to  publish  their  CMaps  on  the  Internet.  The 
software  also  allows  incorporating  resources  (photos,  images,  graphs,  videos,  charts,  tables, 
texts,  internet  pages  or  other  CMaps)  that  can  be  associated  with  concepts  or  linking  words  in  a 
CMap.  Links  to  these  resources  are  displayed  as  icons  underneath  the  concepts  (see  Figure 
1-2). 
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CmapTools  provides  extensive  support  for  collaborative  work  during  CMap  construction.  The 
high  degree  of  explicitness  of  CMaps  makes  them  an  ideal  vehicle  for  the  exchange  of  ideas  or 
for  the  collaborative  construction  of  new  knowledge. 
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2  STRUCTURE  OF  THE  WORKSHOP 

This  section  describes  the  workshop  participants  and  organizers,  the  agenda,  and  the  structure 
of  the  workshop. 


2.1  Introduction 

To  investigate  the  potential  utility  of  CMap  knowledge  models  and  CMapping  for  intelligence 
analysts,  DRDC  Toronto  hosted  a  half-day  workshop  to  generate  feedback  from  intelligence 
community  SMEs.  The  feedback  concerned  the  organization,  design  and  the  potential 
application  of  CMapping,  and  any  implementation  issues  and/or  suggestions  for  improvements. 
The  results  of  the  workshop  will  help  guide  potential  application  of  CMapping  to  intelligence 
activities,  as  well  as  analyst  training. 


2.2  Workshop  Organizers 

The  workshop  was  organized  by  members  of  TRIG  from  DRDC  Toronto  and  CAE  PS  (see 
Table  2-1). 


Table  2-1:  Workshop  Organizers 


Name 

Position 

David  Mandel 

Senior  Defence  Scientist,  Group  Leader  of  TRIG,  and  Workshop 
Organiser,  DRDC  Toronto 

Natalia  Derbentseva 

Natural  Sciences  and  Engineering  Council  of  Canada  Visiting  Fellow 
and  Workshop  Organiser,  DRDC  Toronto 

Tab  Lamoureux 

Senior  Consultant,  CAE  PS 

Michelle  Gauthier 

Consultant,  CAE  PS 

2.3  Participating  Organizations 

Ten  intelligence  SMEs  from  several  government  departments  and  one  industrial  partner 
attended  the  workshop  (see  Table  2-2). 

Table  2-2:  List  of  Participating  Organizations 


Organization 

Organization 

Campbell  Intel  Services  Inc. 

CDI 
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Organization 

Organization 

Canadian  Border  Services  Agency  (CBSA) 

Library  of  Parliament 

Canadian  Forces  School  of  Military  Intelligence 
(CFSMI) 

International  Assessment  Staff  -  Privy  Council 
Office  (IAS-PCO) 

2.4  Agenda 

The  workshop  followed  the  agenda  presented  in  Table  2-3. 


Table  2-3:  Workshop  Agenda 


Time 

Title 

13:00-13:15 

Registration 

13:15-13:45 

Introduction  to  CMapping  and  CMap  knowledge  model 
demonstration 

13:45-14:40 

Breakout  group  discussion  session  1 

14:40-14:50 

Group  reports 

14:50-15:10 

Break 

15:10-16:05 

Breakout  group  discussion  session  2 

16:05-16:15 

Group  reports 

16:15-16:30 

Conclusions  and  Wrap  Up 

2.5  Workshop  structure 

The  following  subsections  provide  details  regarding  the  review  of  CMKMIA,  break-out  sessions, 
and  discussion  activities. 


2.5.1  Introduction  to  CMapping  and  CMKMIA 

The  workshop  began  with  welcoming  and  introductory  remarks  by  Dr.  David  Mandel  followed  by 
an  introduction  to  CMapping  and  an  overview  of  CMap  properties  by  Dr.  Natalia  Derbentseva. 
The  CMKMIA  was  presented  next  to  facilitate  the  consideration  of  CMapping  as  an  effective 
means  of  organizing  information  and  managing  resources. 

The  utility  of  CMap  knowledge  modelling  methodology  for  information  management  and 
education  purposes,  and  as  a  venue  for  discussion  and  collaboration,  was  demonstrated.  The 
potential  breadth  (i.e.  showing  the  variety  of  topics)  and  depth  (i.e.  show  the  level  of  detail  of  a 
specific  map)  of  the  model  were  also  considered. 
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2.5.2  Break-out  Discussion  Sessions 

Two  sets  of  interactive,  group-oriented,  break-out  sessions  were  carried  out.  The  break-out 
groups  were  organized  into  three  concurrent  50-minute  sessions  in  which  teams  pursued  two 
discussion  activities  to  address  several  objectives  related  to  CMapping.  Each  group  was  pre¬ 
assigned  two  workshop  participants  to  fill  the  roles  of  the  rapporteur  and  facilitator.  One  of  the 
workshop  organizers  was  assigned  to  each  of  the  groups  as  an  observer  and  note  taker.  The 
break-out  sessions  provided  participants  with  an  opportunity  to  share  their  thoughts, 
perspectives,  and  reflections  on  CMapping,  and  to  discuss  future  applications. 


2.5.3  Group  Assignment 

Participants  were  assigned  to  one  of  three  groups.  The  groups  were  organized  to  establish 
equal  composition  of  operationally  focused,  international/domestic,  training,  and  academic 
perspectives  (see  Table  2-4). 


Table  2-4:  List  of  Group  Assignments 


Group  1 

Group  2 

Group  3  1 

IAS 

CDI 

CBSA 

CDI 

Campbell  Intel  Services  Inc. 

Library  of  Parliament 

DRDC  (D.  Mandel) 

IAS  -  PCO 

CDI 

CFSMI 

CAE  PS  Canada  (note  taker) 

CDI 

CAE  PS  Canada  (note  taker) 

DRDC  (note  taker) 

2.5.4  Break-out  Session  Questions 

Several  activities  and  questions  related  to  CMapping  were  addressed  and  discussed  during  the 
break-out  sessions.  The  following  section  outlines  the  discussion  activities  and  related 
questions  addressed  during  the  break-out  sessions. 


2. 5.4.1  Discussion  Activity  #1 


Objective  1 

•  What  do  you  see  as  the  potential  uses  of  a  CMap  knowledge  model  or  CMapping  in  general 
within  the  intelligence  community? 

•  Prioritize  the  potential  uses  listed  from  1  (most  important)  onward. 
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Objective  2 

•  Given  your  responses  to  Objective  1 ,  what  benefits,  if  any,  does  CMapping  offer  over  what 
you  or  those  you  supervise  (or  have  supervised)  currently  do? 

•  What  drawbacks,  if  any,  would  you  anticipate  if  CMaps  were  to  be  used? 

•  If  possible,  specify  the  benefits  and  drawbacks  in  relation  to  the  potential  uses  you 
described  in  your  answer  to  Objective  1 . 

2. 5.4.1  Discussion  Activity  #2 

Objective 

•  Start  building  a  CMap  to  address  the  focus  question: 

“ What  is  analytic  integrity?" 

•  First,  generate  a  list  of  the  most  relevant  concepts  that  would  help  answer  the  question.  We 
suggest  you  list  between  6-12  concepts. 

•  Next,  start  connecting  the  concepts  with  specific  linking  phrases  to  form  a  CMap. 

•  Reflect  on  the  experience  of  collaborative  CMap  construction  that  you  just  engaged  in  and 
note  any  pertinent  observations. 

•  What  parts,  if  any,  were  particularly  challenging  and/or  revealing? 

2.5.5  Group  Reports 

Following  each  break-out  discussion  session,  the  main  points  of  the  discussion  were  presented 
to  all  workshop  participants  by  the  pre-assigned  rapporteur  from  each  group. 

2.5.6  Conclusions  and  Wrap-up 

At  the  end  of  the  workshop,  participants  were  encouraged  to  share  their  thoughts,  perspectives, 
and  reflections  on  CMapping,  and  to  discuss  its  future  applications. 
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3  SUMMARY  OF  WORKSHOP  BREAK-OUT  GROUP  SESSIONS 

This  section  details  the  results  of  the  team  discussion  activities  which  addressed  the  objectives 
described  in  Section  2.5.4  and  is  organized  by  the  discussion  activity,  the  objective  and  the 
discussion  results  for  each  team. 


3.1  Discussion  Activity  #1 


3.1.1  Objective  1  -  Potential  Uses  of  CMaps 


3. 1.1.1  Group  1 

Below  is  a  list  of  potential  applications  that  were  identified  and  discussed  by  Group  1 . 

•  Management;  understanding  and  making  adjustments  to  organize  information  (e.q.  tagging 

info):  CMapping  was  seen  as  useful  for  taking  different  types  of  information  and  tagging  that 
information  to  relevant  concepts  that  could  be  tracked  over  time. 

•  Collection  planning;  CMapping  may  be  useful  for  spreading  out  the  argument  or  analysis  in 
a  tree  branching  way  rather  than  using  a  checklist.  The  advantage  of  this  approach  is  that  it 
allows  the  analyst  to  better  match  the  analysis  to  the  end  requirement. 

•  Quality  control:  All  participants  in  the  group  saw  CMapping  as  a  potentially  effective  way  to 
validate  an  argument  and  organize  corroborative  evidence.  The  ability  of  a  CMap  to  support 
the  identification  of  gaps  and  anomalies  in  the  argument  (i.e.  sanity  check),  or  lack  thereof, 
can  help  qualify  the  strength  of  an  argument.  It  allows  the  analyst  to  represent  what  would 
normally  be  a  paragraph  in  visual  form  which  can  be  used  to  more  easily  verify  that  all  the 
information  and  concepts  are  included  in  the  argument.  A  benefit  of  this  approach  is  that  it 
can  be  used  at  a  tactical  level  to  test  the  logic  of  people’s  arguments  which  would  satisfy  the 
burden  of  proof  that  lawyers  need  before  they  give  the  go-ahead  to  further  research  a 
specific  target. 

•  Imagination  (structured  analytical  methodology):  Some  participants  proposed  that  CMaps 
can  be  used  as  an  “imaginative  tool”  for  validating  an  argument  in  a  graphical  form. 

•  Collaboration  tool.  A  benefit  of  CMapping  is  its  ability  to  schematically  lay  out  an  argument. 
This  allows  an  argument  to  become  transparent  and  be  shared  with  others.  This  facilitates 
the  use  of  CMapping  as  a  collaborative  tool.  For  instance,  a  map  could  be  kept  on  a  server 
where  people  could  add  comments  to  the  map  and  share  information.  A  potential 
application  of  the  tool  would  be  with  the  IACC  (Intelligence  Assessment  Coordination 
Committee)  where  different  parts  of  the  intelligence  community  could  come  together  and 
collaborate  and  use  the  different  parts  of  a  CMap  knowledge  model  as  areas  of  discussion. 
As  a  collaboration  tool,  it  can  also  be  used  to  foster  discussion  among  analysts.  During 
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these  discussions,  there  was  some  dissonance  regarding  the  practical  application  of 
CMapping  as  a  collaboration  tool. 

•  Teaching  tool:  Since  CMapping  can  be  used  for  collaboration,  it  could  potentially  be  useful 
as  a  teaching  tool,  and  as  a  means  of  working  through  an  argument  within  a  group 
environment.  This  approach  could  teach  students  to  use  a  structured  approach  to  analysis 
One  participant  meanwhile  suggested  that  intelligence  analyst  training  could  be  used  as  a 
test  bed  to  validate  the  effectiveness  of  the  CmapTools  and  the  CMapping  process. 

Following  the  discussion,  the  group  agreed  on  the  following  potential  uses  of  CMapping  which 
were  presented  during  the  group  report: 

•  Management;  understanding  and  making  adjustments  to  organize  information  (e.g.  tagging 
info); 

•  Collection  planning; 

•  Quality  control; 

•  Teaching  tool; 

•  Imagination  (structured  analytical  methodology);  and 

•  Collaboration  tool. 

It  was  agreed  by  all  participants  in  the  group  that  they  could  not  prioritize  the  potential  uses 
identified  because  their  use  depends  on  the  context  in  which  CMapping  is  applied.  The  group 
did  reach  a  consensus  however,  whereby  quality  control,  an  imagination  tool  and  collaboration 
were  likely  to  be  equally  useful. 


3.1. 1.2  Group  2 

Below  is  a  list  of  potential  applications  that  were  identified  and  discussed  by  Group  2. 

•  Explore  a  topic  in  depth:  Because  the  approach  relies  on  a  systematic  drill  down  into 
concepts  and  all  potential  relationships,  it  is  possible  to  quickly  reach  a  deep  level  of 
understanding  about  a  topic  very  quickly. 

•  Structure  the  intelligence  problem:  CMapping  can  be  used  as  an  aid  to  structure  the 
intelligence  problem. 

•  Strategic  tool  to  discipline  the  thinking  process:  This  approach  objectifies  and  externalizes 
the  analytical  process  which  can  be  useful  for  disciplining  the  thinking  process.  Participants 
recognized  that  structuring  and  framing  the  analysis  is  a  benefit  for  this  approach.  It  was 
noted  that  it  takes  time  to  create  a  CMap  and  may  therefore  not  be  appropriate  for 
addressing  certain  issues  that  require  a  quick  assessment  of  the  problem  scope.  The 
benefits  of  CMaps  must  therefore  be  weighed  against  the  drawbacks  when  evaluating  the 
added  value  of  using  CMaps. 
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•  Managing  analysts:  CMapping  can  help  identify  assumptions  which  can  be  used  to  manage 
analysts’  analytical  process. 

•  Validation  mechanism:  The  externalization  of  the  process  increases  the  transparency  of  the 
results  which  can  be  used  for  verification  purposes  to  assess  the  completeness  of  the 
analysis  that  is  performed. 

•  Audit  and  defensibility:  The  representation  of  the  process  followed  by  the  analyst,  including 
their  thinking,  allows  for  downstream  audit  of  the  analysis  to  ensure  the  guality  of  the 
analyses  performed  by  the  intelligence  organization.  The  clear  representation  of  both  the 
analysis  and  the  thinking  the  analyst  brought  to  the  analysis  permits  the  decisions  made  by 
the  intelligence  organization  to  be  defended  at  a  later  date,  should  they  be  challenged  by  an 
oversight  review  or  commission  of  inguiry. 

•  Collaboration  Tool:  The  greater  transparency  of  CMaps  also  allows  individuals  to  guery  the 
ideas  and  assumptions  held  by  others.  This  dialogical  approach  can  provoke  discussion 
which  can  promote  a  shared  understanding  by  rendering  ideas  and  assumptions  of 
individuals  transparent,  and  allowing  others  to  guery  and  understand  them. 

•  Securing  additional  resources:  The  CMap  can  demonstrate  to  the  consumer  the  breadth  of 
analysis  that  has  led  to  a  conclusion.  It  can  also  help  decision  makers  understand  what 
resources  (in  terms  of  knowledge  and  experience)  are  involved  in  an  analysis.  This  can 
then  be  used  to  persuasively  argue  for  additional  resources  to  overcome  deficiencies  in  time 
or  potential  decreases  in  guality.  A  disadvantage  of  this  approach  is  that  the  specific 
deficiencies  cannot  be  weighted  using  the  CmapTools. 

•  Visual  tool:  CMapping  provides  a  visualization  of  the  relationship  between  ideas  which  is 
guicker  than  reading  lots  of  pages.  This  allows  the  analyst  to  have  an  overview  (top  down 
approach)  of  the  problem  scope.  A  graphic  representation  of  the  problem  space  can  be 
useful  for  some  decision  makers  who  have  limited  time  available  to  ‘read  in’  to  a  problem 
and  who,  anecdotally,  tend  to  share  a  visual  approach  to  rapid  comprehension.  The  ability 
to  arrange  words  spatially  and  linking  them  in  a  visual  form  can  provide  a  higher  intensity 
stimulus  that  can  be  used  for  presentation  purposes.  This  presentation  can  be  adapted  for 
the  recipient,  such  as  the  analyst  creating  the  map,  the  approver  of  an  analysis,  or  the 
consumer. 

•  Training  tool  to  accelerate  learning:  CMapping  was  seen  to  be  a  good  training  tool  to  convey 
a  higher  level  of  abstraction  regarding  a  particular  intelligence  issue.  A  CMap  created  by  an 
expert  desk  analyst  may  be  used  as  a  training  aid  for  a  novice  analyst  taking  over  the  desk, 
which  may  accelerate  his  or  her  learning.  The  expert’s  CMap  may  be  adapted  and 
augmented  over  time.  The  benefit  of  this  approach  in  training  is  that  it  is  creative  and  seen 
to  be  suited  for  the  new  generation  of  students. 

Following  the  discussion  activity,  the  group  listed  out  four  potential  uses  of  CMapping  and 

prioritized  them  in  descending  order.  These  were  presented  to  the  workshop  participants  during 

the  group  report  session. 

1.  Structure  the  intelligence  problem; 

2.  Act  as  a  training  tool/model; 

29  June  2010  -  11  -  5035-001  Version  04 

©  Her  Majesty  the  Queen  as  represented  by  the  Minister  of  National  Defence,  2010 
©  Sa  majeste  la  reine,  representee  par  le  ministre  de  la  Defense  nationale,  2010 


CAE 


Validation  Workshop  of  the 
DRDC  Concept  Map  Knowledge  Model: 

Issues  in  Intelligence  Analysis 


3.  Discipline  the  analysis  process;  and 

4.  As  a  tool  for  securing  additional  resources. 


3. 1.1. 3  Group  3 

Below  is  a  list  of  potential  applications  that  were  identified  and  discussed  by  Group  3. 

•  Foster  group  discussion:  CMapping  was  seen  as  useful  at  the  beginning  of  an  analysis  to 
foster  discussion  among  analysts  by  identifying  possible  concepts  and  sub-concepts 
involved  in  an  argument.  It  can  also  be  used  at  the  end  of  the  analysis  as  a  means  to  verify 
that  everyone  has  a  common  understanding  of  the  problem  space. 

•  An  analytical  tool  to  structure  an  analysis  and  to  analyse  the  organization  structure: 
Participants  recognized  that  the  tool  would  be  useful  to  structure  the  analytical  process  as  it 
allows  an  analyst  to  decompose  the  process  into  discrete  steps,  and  to  break  down  a 
problem  into  concepts  and  sub-concepts.  The  benefit  of  this  approach  is  that  it  visually 
captures  very  complex  thinking  and,  for  instance,  will  quickly  show  illogical  arguments.  The 
ability  of  CMapping  to  break  down  a  problem  into  concepts  and  sub-concepts  allows  an 
interested  party  to  analyze  the  organizational  structure  of  that  analysis  and  identify  leverage 
points  within  a  system  (one  of  the  participants  referred  to  this  process  as  a  “system  of 
systems  analysis”). 

•  Visualization  (visual  tool):  All  participants  agreed  that  CMapping  provided  a  visual 
representation  of  the  analysis  and  externalized  one’s  logical  thinking.  This  externalization 
allows  an  analyst  (or,  more  likely,  another  analyst)  to  identify  what  is  missing  in  the 
argument  or  certain  gaps  in  logic.  It  was  recognized,  however,  that  this  method  may  only  be 
appropriate  for  “visual  thinkers”.  For  these  types  of  people,  gaps  in  logic  may  be  harder  to 
find  in  prose  rather  than  in  a  visual  map. 

•  Validation:  All  participants  in  the  group  saw  CMapping  as  an  effective  way  to  validate  an 
argument  by  looking  back  on  one’s  own  work  and  identifying  what  is  missing,  what  gaps 
there  are,  and  what  should  be  explored  further. 

•  Prioritization  of  collection  management:  One  participant  suggested  that  CMapping  could  be 
used  to  identify  what  information  has  already  been  acquired  and  what  is  already  known 
about  a  particular  topic.  Once  identified,  analysts  could  then  identify  and  collect  information 
that  is  missing  or  unknown. 

•  Resource  forecasting.  One  participant  suggested  that  CMapping  could  be  used  to  identify  a 
range  of  probabilities  which  could  be  developed  into  a  decision  tree.  This  would  be  useful 
for  resource  forecasting  exercises  and  also  for  expanding  the  minds  of  decision  makers. 

Following  the  discussion  activity,  the  group  selected  four  potential  uses  of  CMapping  and 

prioritized  them  in  descending  order.  These  were  presented  to  the  workshop  participants  during 

the  group  report  session. 

1 .  Prompts  discussion:  process  of  putting  together  CMap  and  process  of  CMapping; 
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2.  Structure  and  analysis:  process  of  structuring  the  analysis,  decomposing; 

3.  Visualization:  having  done  analysis,  can  look  at  it,  ability  to  show  smart  work;  and, 

4.  Validation:  looking  back  on  your  work:  what  is  missing,  what  gaps  are  there,  where  should 
we  explore  further? 


3.1.2  Objective  2  -  Benefits  and  drawbacks  of  CMaps 
3. 1.2.1  Group  1 

Below  is  a  list  of  potential  benefits  and  drawbacks  that  were  discussed  by  Group  1.  Some  of 
these  were  discussed  in  relation  to  the  potential  uses  of  CMapping  (see  Section  3. 1.1.1  above). 


Benefits 

•  CMapping  can  address  the  “systems  question”  (the  big  picture)  to  visualize  the  inter¬ 
relationships  between  concepts  and  see  how  the  bits  and  pieces  of  intelligence  analysis  fit 
together.  CMapping  allows  more  focus  on  the  issue  as  it  would  allow  the  analyst  to  reason 
through  their  thought  process,  to  focus  their  research  and  in  problem  solving.  In  this  way, 
CMapping  would  be  useful  as  an  analytical  and  validation  tool. 

•  CMaps  can  support  the  identification  of  gaps  and  anomalies  in  the  argument  which  is  useful 
for  validating  arguments.  CMapping  allows  the  thought  process  to  be  clearly  articulated  by 
schematically  laying  out  an  argument.  A  reviewer  of  a  CMap  can  follow  the  logic  of  an 
author’s  arguments.  In  this  way,  a  CMap  would  be  useful  as  a  collaboration  and  validation 
tool. 

•  CMapping  is  a  simple  process  (only  some  participants  saw  this  benefit). 

•  CMapping  is  a  good  methodology  for  analysts  with  a  ‘visual’  style  of  thinking. 

•  CMapping  can  make  a  person  think  by  creating  dissonance. 

•  CMapping  can  offer  the  analyst  lots  of  options  and  answers.  For  instance,  CMapping  can 
be  used  as  a  flowchart  to  identify  data  that  goes  into  the  argument.  Options  can  then  be 
abstracted  from  the  data. 

•  CMapping  can  be  a  time  saver  (without  wasting  time  writing  it  on  paper  first).  The  process 
of  CMapping  was  recognized  by  some  participants  to  be  a  time  saver  during  the  process  of 
intelligence  analysis  as  it  enables  a  person  to  brainstorm  their  ideas  and  form  them  into  a 
complete  argument  prior  to  putting  their  ideas  into  paragraph  form. 

•  CMaps  have  the  ability  to  link  propositions  in  an  explicit  manner.  CMapping  is  linked  to 
language  in  the  sense  that  it  is  propositional,  but  it  is  not  constrained  by  the  linear  formation 
of  sentences. 

•  CMaps  can  allow  the  analyst  to  better  match  the  analysis  to  the  end  requirement. 
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•  CMaps  are  good  to  foster  discussion  and  can  be  useful  as  a  collaboration  tool.  For 

instance,  CmapTools  allows  people  to  share  information  over  a  server  or  over  the  internet. 


Drawbacks 

•  CMapping  is  not  good  for  discrete  questions,  targeting  specific  events. 

•  An  analyst  with  a  ‘linear’  style  of  thinking  will  get  lost  in  the  lines  of  a  CMap.  It  is  not 
appropriate  for  all  cognitive  styles.  That  is,  it  was  noted  by  some  participants  that  this 
approach  may  only  be  a  good  methodology  for  people  who  are  non-linear,  visual,  creative 
thinkers.  One  participant  suggested  that  linear  kinds  of  thinkers  would  get  bogged  down  in 
the  model  and  lose  track  of  the  relationships  between  concepts. 

•  The  answer  is  not  readily  clear  in  a  CMap.  In  this  way,  it  may  be  difficult  to  use  as  an 
analytical  tool. 

•  It  takes  time  to  learn  the  method  of  CMapping  and  CmapTools.  Analysts  may  therefore 
spend  their  energy  on  learning  the  tool  rather  than  on  the  specific  issue  at  hand  (only  one 
participant  argued  this). 

•  CMapping  is  not  practical  in  the  field  (only  one  participant  argued  this).  He  argued  that  the 
complex  and  time-consuming  process  of  CMapping  would  not  be  practical  in  light  of  the  day- 
to-day  pressure  of  producing  analytic  work  against  time  constraints. 

•  CMapping  and  CmapTools  need  validation  as  they  are  still  in  the  experimental  stage. 
CMapping  could  be  tested  during  the  training  curriculum  to  demonstrate  how  it  might 
actually  be  used  in  the  field.  It  should  be  noted  that  one  participant  argued  that  not  all  new 
processes  and  tools  can  be  tested  prior  to  use. 

•  CMapping  is  a  complicated  process  (only  one  participant  expressed  this  drawback). 

•  CmapTools  is  yet  another  tool  (similar  to,  for  example,  i2). 

•  CmapTools  is  not  a  presenting  tool.  Although  it  may  be  a  good  method  to  restructure  and 
present  ideas,  the  CMap  would  need  to  be  restructured  to  outline  the  major  discussion 
points  in  order  to  present  it  to  others. 

•  A  CMap  is  not  self  explanatory.  It  is  difficult  to  find  the  entry  point  and  the  answer  and  may 
therefore  not  be  practical  in  all  types  of  analyses. 

Following  the  discussion  related  to  Objective  2,  the  following  benefits  and  drawbacks  of 

CMapping  were  reported  by  the  rapporteur  from  Group  1  during  the  group  report  activity. 


Benefits 

•  Can  address  the  “systems  question”  (the  big  picture). 

•  Good  for  analysts  with  a  ‘visual’  style  of  thinking. 

•  Gives  lots  of  options/answers. 
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•  Can  be  a  time  saver  (without  wasting  time  writing  it  on  paper  first). 


Drawbacks 

•  Not  good  for  discrete  questions. 

•  Linear  analyst  will  get  lost  in  the  lines. 

•  Answer  is  not  readily  clear. 

•  Takes  time  to  learn  the  method  and  the  tool. 

•  Needs  validation  as  it  is  still  in  the  experimental  stage. 

•  Yet  another  tool  (e.g.  i2). 


3. 1.2.2  Group  2 

Below  is  a  list  of  potential  benefits  and  drawbacks  that  were  discussed  by  Group  2.  Some  of 
these  were  discussed  in  relation  to  the  potential  uses  of  CMapping  (see  Section  3. 1.1. 2  above). 


Benefits 

•  The  graphical  approach  of  a  CMap  acts  as  a  visual  checklist  of  concepts.  This  benefit 
would  make  a  CMap  useful  for  structuring  the  analysis,  training  and  to  request  additional 
resources. 

•  CMapping  and  CMaps  allow  for  a  deeper  understanding  of  a  topic  by  looking  at  all  potential 
relationships  and  would  therefore  be  useful  to  explore  a  topic  at  depth. 

•  The  graphical  approach  of  a  CMap  allows  a  quick  assessment  of  the  whole  problem  space. 
It  is  quicker  than  reading  lots  of  pages  (top  down  approach). 

•  CMapping  objectifies,  makes  explicit  and  externalizes  the  analysis  process  which  would  be 
useful  to  discipline  the  thinking  process.  CMapping  as  a  result  makes  the  process  of 
analysis  transparent. 

•  A  CMap  identifies  the  completeness  of  an  argument  by  identifying  what  is  known  and  what 
is  unknown  and  would  therefore  be  useful  as  a  validation  tool. 

•  A  CMap  provokes  conversation  and  may  promote  a  shared  understanding. 

•  A  CMap  can  also  help  decision  makers  understand  what  resources  (in  terms  of  knowledge 
and  experience)  are  involved  in  an  analysis.  A  CMap  could  be  useful  as  a  tool  to  request 
additional  resources. 

•  CMapping  allows  for  a  higher  level  of  abstraction  for  problem  solving. 

•  A  CMap  can  be  adapted  for  the  recipient,  such  as  the  analyst  creating  the  map,  the 
approver  of  an  analysis  and  the  consumer. 
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•  CMapping  is  an  unconstrained  process.  Since  problems  are  not  linear,  but  fluid  and 

dynamic,  CMapping  may  be  better  suited  than  traditional  linear  problem  solving  to  deal  with 
the  nature  of  complex  intelligence. 


Drawbacks 

•  CMapping  is  not  for  non-visual  thinkers. 

•  The  greater  the  complexity  of  the  problem,  the  more  complex  the  CMap  will  be.  A  linear 
method  may  be  more  appropriate  for  highly  complex  problems. 

•  The  focus  question,  if  not  formed  well,  can  be  a  constraint  on  how  well  the  CMap  represents 
a  problem  and  how  well  it  facilitates  the  group’s  ability  to  explore  a  problem. 

•  There  is  no  weighting  factor  in  CmapTools.  It  is  not  possible  to  show  the  relative  importance 
of  certain  concepts. 

•  The  answer  is  not  readily  available  in  a  CMap  and  it  does  not  necessarily  help  resolve 
complex  problems.  While  the  CMaps  help  to  tease  out  the  elements  of  the  problem  space, 
it  does  not  necessarily  provide  the  “so  what?”  or  the  conclusion. 

•  There  is  no  guarantee  that  the  process  of  CMapping  will  lead  to  a  better  understanding  or 
better  outcome.  There  is  no  measurement  of  how  close  the  CMap  is  to  the  reality  of  the 
problem.  It  is  therefore  unclear  where  the  value  of  CMapping  would  provide  a  better 
outcome  over  other  approaches. 

•  The  process  of  CMapping  is  time  consuming. 

•  There  are  not  a  lot  of  drawbacks  in  the  training  context,  except  that  some  people  may  see  it 
as  overly  ‘arty’. 

Following  the  discussion  related  to  Objective  2,  the  following  benefits  and  drawbacks  of 

CMapping  were  reported  by  the  rapporteur  from  Group  2  during  the  group  report  activity. 


Benefits 

•  Visual  (quicker). 

•  Checklist:  review  quickly. 

•  Appreciation  of  problem  space. 

•  Consideration  of  complex  problems  (instead  of  linear  problems). 

•  Links  words  with  spatial  representation  -  how  far  concepts  are  related  /  non-related. 

•  Educational  -  audience  and  students. 
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Drawbacks 

•  Focus  question  itself  if  not  phrased  well. 

•  Strategically  not  tactical  (where  need  quick  turnaround). 

•  It  is  part  of  a  toolbox  (not  a  panacea). 

•  Measure  relative  quality  of  outcome. 

•  Lack  of  weighting. 

•  Not  guarantee  of  better  outcome  (inputs,  propositions,  may  be  delusional). 

3. 1.2. 3  Group  3 

Below  is  a  list  of  potential  benefits  and  drawbacks  that  were  discussed  by  Group  3.  Some  of 
these  were  discussed  in  relation  to  the  potential  uses  of  CMapping  (see  Section  3. 1.1. 3  above). 


Benefits 

•  CMaps  can  identify  capabilities  within  a  system  (i.e.  system  of  systems  analysis)  by 
decomposing  the  process  into  discrete  steps  and  breaking  down  a  problem  into  concepts 
and  sub-concepts.  This  makes  them  useful  for  analyzing  the  organizational  structure  for 
any  analytic  challenge. 

•  CMapping  can  capture  visually  very  complex  thinking  (if  an  argument  is  not  logical,  then  this 
will  quickly  be  shown  in  a  CMap).  In  this  way,  a  CMap  is  a  powerful  tool  for  identifying  what 
is  missing  and/or  gaps  in  logic. 

•  CMapping  externalizes  thinking  and  identifies  strength  of  arguments.  This  makes  CMaps 
useful  for  identifying  how  the  analysis  should  be  further  explored. 

•  CMaps  help  analysts  and  the  audience  get  the  full  picture. 

•  CMaps  can  help  communicate  ideas. 

•  CMapping  is  good  for  visual  people.  For  these  people,  a  CMap  may  make  it  easier  to 
manage  an  analysis  than  when  it  is  in  a  textual  form. 

•  A  CMap  can  identify  a  range  of  probabilities  which  would  be  useful  in  forecasting  exercises. 
A  decision  tree  can  be  created  from  a  CMap  and  used  to  support  the  thinking  of  the  decision 
makers  and  open  up  a  greater  range  of  possibilities. 

•  CMapping  answers  conceptual  or  systemic  questions.  For  example,  tangible  systemic 
questions  such  as  “What  is  the  nature  of  (Hamid)  Karzai’s  (President  of  Afghanistan) 
power?”  is  well-suited  to  CMapping. 

•  CMaps  link  concepts.  CMapping’s  ability  to  identify  concepts  that  connect  to  one  another  is 
additional  powerful  methodology  that  could  be  added  to  the  existing  repertoire  of  tools. 
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CMaps  are  therefore  useful  for  visualizing  propositional  relationships.  The  use  of  verbs  as 
descriptions  within  propositions  was  seen  as  the  real  benefit  with  CMapping. 


Drawbacks 

•  CMaps  are  not  easy  to  manage  for  people  who  are  not  visual  thinkers.  CMapping  may  not 
be  appropriate  for  people  who  are  not  visual  thinkers. 

•  The  CMap  needs  to  be  well  crafted  in  order  to  represent  thinking  well. 

•  It  is  not  as  intuitive  to  create  a  map  of  “what  can  be”. 

•  One  participant  recognized  that  there  are  other  similar  tools  that  are  already  being  used  by 
the  intelligence  community  (e.g.  i2)  and  that  CMaps  may  not  offer  anything  over  what  these 
other  tools  already  do. 

•  One  can  answer  only  a  narrow  range  of  questions  with  a  CMap.  CMapping  would  be  less 
useful  to  answer  less  tangible  or  discrete  questions  like  “What  would  be  the  impact  of 
Karzai’s  death?” 

Following  the  discussion  related  to  Objective  2,  the  following  benefits  and  drawbacks  of 

CMapping  were  reported  by  the  rapporteur  from  Group  3  during  the  group  report  activity. 


Benefit 

•  Verb  use  (can  use  any  verb  as  description  within  the  proposition). 


Drawbacks 

•  Narrow  range  of  questions  that  can  be  answered. 

•  Cannot  answer  the  following  question:  “What  would  the  impact  of  Karzai’s  death  be?” 


3.2  Discussion  Activity  #2 


3.2.1  Objective  -  CMap  Construction 

Participants  were  given  a  limited  amount  of  time  to  construct  a  CMap  of  their  own  and  to  form 
impressions  about  the  process  of  building  a  CMap  and  its  possible  utility  to  intelligence  analysis 
During  the  workshop  groups  generated  their  maps  on  paper  flip  charts.  These  maps  were  later 
transferred  into  an  electronic  format  using  CmapTools.  The  CMap  outputs  generated  by  each 
group  are  presented  in  the  following  sections. 
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3.2. 1.1  Group  1  -  CMap  of  Analytic  Integrity 

Below  is  Group  1’s  CMap  of  analytic  integrity  following  the  group’s  discussion  activity  (see 
Figure  3-1).  The  CMap  in  Figure  3-1  was  created  with  CmapTools  by  the  author  following  the 
workshop.  The  original  CMap  drawn  by  Group  1  can  be  seen  in  Annex  A,  Figure  A-1 . 


Figure  3-1 :  Group  1  -  CMap  of  analytic  integrity 


3.2. 1.1.1  Group  1  -  Identification  of  the  challenging  and/or  revealing  parts  of 
CMapping 

The  following  issues  were  identified  by  Group  1  as  particularly  challenging  or  revealing  with 
respect  to  CMapping: 

•  It  is  difficult  to  decompose  a  vague  idea; 

•  CMapping  is  a  difficult  process; 

•  CMapping  can  be  an  effective  tool  for  exploration  and  education  beyond  that  of  current 
methods;  and 

•  The  tool  does  not  have  ability  to  weight  the  propositions.  There  is  a  need  for  a  tool  that 
blends  this  capability. 


3.2.1 .2  Group  2  -  CMap  of  analytic  integrity 

Below  is  Group  2’s  CMap  of  analytic  integrity  following  the  group’s  discussion  activity  (see 
Figure  3-2).  Note  that  some  concepts  were  not  linked  given  the  limited  time  available  to  perform 
the  task.  The  CMap  in  Figure  3-2was  created  with  CmapTools  by  the  author  following  the 
workshop.  The  original  CMap  drawn  by  Group  2  can  be  seen  in  Annex  A,  Figure  A-2,  Figure 
A-3,  and  Figure  A-4. 
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Figure  3-2:  Group  2  -  CMap  of  analytic  integrity 

3.2.1 .2.1  Group  2  -  Identification  of  the  Challenging  and/or  Revealing  Parts  of 
CMapping 

The  following  difficulties  were  identified  by  Group  2  in  terms  of  CMapping: 

•  Definition  of  terms; 

•  Coming  up  with  linking  terms; 

•  Developing  effective  concepts;  and 

•  Overlap/redundancy. 

3.2.1 .3  Group  3  -  CMap  of  Analytic  Integrity 

Below  is  Group  3’s  CMap  knowledge  model  of  analytic  integrity  following  the  group’s  discussion 
activity  (see  Figure  3-3).  Note  that  some  concepts  were  not  linked  given  the  limited  time 
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available  to  perform  the  task.  The  CMap  in  Figure  3-3  was  created  with  CmapTools  by  the 
author  following  the  workshop.  The  original  CMap  drawn  by  Group  3  can  be  seen  in  Annex  A, 
Figure  A-5. 


Figure  3-3:  Group  3  -  CMap  of  analytic  integrity 


3.2.1 .3.1  Group  3  -  Identification  of  the  Challenging  and/or  Revealing  Parts  of 
CMapping 

The  following  difficulties  were  identified  by  Group  3  in  terms  of  CMapping: 

•  How  to  use  active  and  passive  voice; 

•  Uni-directional  arrows  (use  of  multi-directional  arrows  would  have  helped);  and 

•  CMapping  is  very  difficult:  first  have  to  define  the  term  because  it  is  vague. 
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4  DISCUSSION  OF  WORKSHOP  RESULTS 


This  section  details  observations  made  during  the  group  discussion  sessions.  The  first 
subsection  is  organized  in  accordance  with  the  group  and  the  discussion  session.  The 
subsequent  subsections  reflect  a  logical  grouping  of  outcomes  from  the  workshop:  potential 
application  and  benefits  of  CMapping,  implementation  issues  of  CMapping  and  suggestions  for 
improvements. 


4.1  Observations  during  discussion  activity  sessions 

The  discussion  activities  elicited  a  lot  of  comments  and  suggestions  from  the  participants. 
Overall,  participants  were  positive  about  the  potential  use  of  CMapping  for  the  intelligence 
community.  Participants  could  readily  see  many  potential  uses  of  CMapping  in  general  and 
identified  some  benefits  and  drawbacks  in  terms  of  these  uses.  The  following  sub-sections 
detail  the  observations  that  were  captured  during  discussion  activity  #1  and  the  construction  of 
the  CMaps. 


4.1.1  Observations  captured  during  the  discussion  activity  #1 


4.1. 1.1  Group  1 

There  was  some  dissonance  among  participants  in  Group  1  regarding  the  practical  application 
of  CMapping  in  general  and  the  tools  needed  to  create  CMap  knowledge  models  in  everyday 
analysis  tasks.  In  particular,  one  participant  saw  CMapping  as  a  time  consuming  and  complex 
process  that  would  not  be  practical  in  the  field.  Other  participants  in  the  group,  however,  saw 
value  in  CMapping’s  ability  to  structure  and  frame  the  analysis,  particularly  its  ability  to  visualize 
the  relationship  between  concepts.  This  dissonance  led  to  discussions  centred  on  the  practical 
applications  of  CMapping  within  the  intelligence  community  and  the  benefits  and  drawbacks  of 
these  applications. 


4.1. 1.2  Group  2 

Group  2  tended  to  focus  on  characteristics  of  the  CMapping  approach  rather  than  on  the 
application  itself.  There  was  also  an  absence  of  disagreement  between  the  participants,  making 
it  difficult  to  decide  whether  the  views  expressed  by  participants  were  consensual  or  whether 
they  merely  decided  not  to  express  disagreement.  Nevertheless,  a  number  of  potentially  useful 
aspects  and  applications  were  identified. 
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4.1. 1.3  Group  3 

Much  of  the  discussion  during  this  breakout  session  surrounded  the  question  of  “  What  does 
CMapping  offer  over  what  you  do  now  (different)?”  Some  participants  saw  the  process  of 
CMapping  as  very  similar  to  other  processes  that  use  other  tools,  e.g.  i2,  Mind  Manager,  to 
perform  intelligence  analysis.  One  participant  was  familiar  with  Mind  Mapping  as  he  had  used  it 
frequently  to  brainstorm  about  particular  topics  and  to  validate  his  arguments.  This  participant 
recognized  how  CMapping  differed  from  current  processes  in  that  it  allows  the  use  of  verbs  to 
describe  the  propositions.  Mind  Mapping,  for  instance,  is  only  a  hierarchical  representation  of 
an  argument  with  no  qualified  relationships  between  concepts.  Another  participant  argued  that 
i2  does  develop  relationships,  and  even  puts  a  weight  on  the  relationships.  It  was  argued, 
however,  that  CMapping  explains  a  concept,  it  is  a  visual  representation  of  the  “what”  -  which, 
to  the  participant,  seemed  different  from  a  power  relationship  (that  i2  offers  for  instance). 


4.1.2  Observations  captured  during  the  discussion  activity  #2 


4.1. 2.1  Group  1 

During  the  group’s  efforts  to  identify  the  various  core  concepts  that  compose  analytic  integrity,  it 
was  quickly  recognized  by  all  participants  that  CMapping  was  a  difficult  process.  The  group 
began  to  build  a  list  of  the  most  relevant  core  concepts  that  constitute  analytic  integrity. 

During  the  activity,  participants  acknowledged  that  the  resulting  CMap  can  be  used  as  a  way  to 
show  consumers  how  their  conclusions  were  derived.  It  also  occurred  to  the  group  that  the 
process  of  identifying  concepts  and  relationships  enabled  them  to  derive  conclusions  that  would 
not  have  been  as  readily  apparent  if  the  concepts  were  shown  on  a  Microsoft  PowerPoint  slide 
as  bullet  points.  For  instance,  it  was  concluded  that  the  group’s  CMap  of  analytic  integrity 
demonstrated  that  integrity  and  quality  were  the  core  components  anchoring  all  of  the  other 
components  in  the  model.  In  identifying  the  propositional  relationships  between  the  concepts; 
the  group  came  to  realize  that  quality  and  integrity  is  fundamentally  the  same  thing.  This  was 
evidence  to  the  group  that  CMapping  was  indeed  a  “thinking  tool”. 

Most  interestingly,  one  participant  who  had  previously  argued  against  the  practicality  of 
CMapping  for  analytic  tasks  came  to  believe  following  this  exercise  that  “the  structured 
brainstorming  of  CMapping  and  of  decomposing  the  component  parts  may  be  the  first  step  of 
intelligence  analysis”. 


4.1. 2.2  Group  2 

The  following  observations  were  made  by  the  member  of  the  organising  group  who  participated 
in  this  session.  They  are  therefore  opinion-based. 

The  construction  of  the  CMap  was  popular  and  enjoyable  with  the  participants  because  it  was  a 
participative  learning  process,  rather  than  simply  using  Microsoft  PowerPoint.  The  group  had 
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difficulty  with  the  definition  of  the  terms  used,  but  only  once  they  began  to  try  and  create  linking 
phrases  between  concepts.  Prior  to  this  there  was  probably  an  assumption  that  everyone 
shared  an  understanding  about  the  terms.  There  was  a  great  deal  of  overlap  and  redundancy 
among  the  terms. 

The  group  had  great  difficulty  coming  up  with  linking  terms  and  changing  their  perspective  to 
accommodate  the  directional  nature  of  relationships.  A  member  of  another  group  noted 
problems  with  grammar,  and  a  more  standard  approach  to  the  grammar  of  the  linking  term  (e.g. 
use  only  active  tense)  would  certainly  have  addressed  this.  A  related  issue  is  that  the  group 
had  difficulty  reading  the  resulting  CMap,  and  would  often  misread  the  relationship.  Often,  this 
was  a  product  of  the  ‘directionality’  of  the  relationship;  the  group  needed  some  training  and 
experience  to  immediately  see  the  causality  that  is  built  into  the  CMap  representation. 

The  group  had  difficulty  developing  effective  concepts  for  the  map,  as  was  evidenced  by  the 
redundancy.  Again,  this  may  have  been  due  to  their  unconscious  biases,  and  may  also  have 
been  due  to  a  quickly  adopted  focus  on  personality  traits,  rather  than  trying  to  reduce  focus  and 
increase  the  orthogonality  of  the  concepts. 

Overall,  it  is  not  surprising  that  this  group  of  experts  had  difficulty  with  the  task.  The  process  of 
decomposing  an  expert  process  is  very  difficult  for  experts  because  they  have  internalised  the 
skills  and  knowledge  required  to  do  the  job  and  thus  have  difficulty  verbalising  this  information. 
This  group  probably  needed  a  facilitator  with  training  and  experience. 


4.1. 2. 3  Group  3 

The  group  initially  broke  down  analytic  integrity  into  two  component  parts  “Analysis”  and 
“Integrity”  and  a  number  of  concepts  comprising  analytic  integrity  were  derived.  During  the 
decomposition  process,  the  group  defined  the  term  ‘analytical  integrity’  in  order  to  help  them 
more  clearly  identify  the  concepts  of  analytic  integrity.  While  participants  were  trying  to  define 
analytic  integrity,  they  came  to  realize  that  the  very  process  of  brainstorming  its  concepts  was  in 
itself  providing  them  with  the  definition.  The  following  concepts  were  derived  by  the  group: 
ethics,  rigor,  bias,  assumptions,  training,  directives  (mission),  policy,  intelligence  problem,  and 
political  command  pressures. 

The  group  then  began  to  focus  on  connecting  the  various  concepts  with  specific  linking  phrases 
to  form  a  CMap.  It  was  during  the  process  of  creating  propositions  that  the  group  ran  into 
difficulties  in  constructing  the  map. 

•  The  group  had  difficulty  creating  propositions  without  a  more  precise  definition  of  analytic 
integrity.  The  following  definition  was  proposed:  “it  is  a  conscious  decision  to  not  ignore  or 
change  evidence  or  to  change  conclusions  because  of  external  pressures”.  This  more 
concrete  definition  provided  a  context  in  which  participants  could  work  out  the  relationships 
among  the  concepts. 

•  There  was  some  difficulty  among  the  group  participants  in  determining  if  an  active  or  a 
passive  voice  should  be  used  to  word  the  propositions.  Participants  concluded  that  a 
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passive  voice  shows  the  relationship  in  the  opposite  direction  of  the  visual  arrow  (they  are 
uni-directional).  Participants  interpreted  the  direction  of  the  arrow  in  a  CMap  to  imply  the 
direction  of  causality  in  the  relationship,  which  caused  a  great  difficulty  in  accepting  the 
possibility  of  linking  phrases  in  a  passive  voice.  Therefore,  it  was  concluded  that  active 
wording  should  be  used  for  the  propositions. 

•  Participants  did  not  realize  that  the  arrows  could  be  directed  upwards  as  well  as  downwards. 
This  led  to  some  confusion  as  to  how  to  structure  the  CMap. 

Due  to  these  difficulties  the  group  did  not  advanced  as  much  in  linking  concepts  in  their  CMap 
as  the  other  two  groups.  Interestingly,  the  resulting  map  following  the  activity  contained  almost 
none  of  the  original  concepts  that  were  generated  at  the  beginning  of  the  exercise.  The  group 
concluded  that  constructing  a  CMap  is  a  challenging  task.  The  passive  wording  of  the 
propositions  made  the  task  particularly  difficult.  One  participant  made  a  final  comment  that  one 
would  need  training  to  effectively  do  CMapping. 


4.2  Potential  applications  and  benefits  of  CMapping 


4.2.1  Potential  applications  of  CMapping 

The  potential  uses  that  were  derived  during  the  break-out  session  discussions  for  all  three 
groups  are  tabulated  into  Table  4-1  below.  As  can  be  seen  in  the  table,  there  is  much  overlap 
between  the  groups  in  regards  to  the  potential  applications  of  a  CMap  knowledge  model  or 
CMapping  in  general  for  the  intelligence  community.  This  is  an  interesting  finding  because  all 
three  groups  arrived  at  these  conclusions  separately  during  the  break-out  sessions.  It  is  also  an 
interesting  outcome  because  each  group  was  composed  of  different  organizations  within  DND 
and  other  government  departments. 

Table  4-1:  List  of  potential  uses  for  CMapping  derived  by  the  groups 


Group  1 

Group  2 

Group  3 

“Imaginative  tool”  for  validating  an 
argument  in  a  graphical  form 

Visual  tool  to  see  all  concepts  at  a 
glance;  visual  approach  to  thinking 

Visualization  /  visual  tool 

Thinking  tool  to  structure  the 
thinking  process 

Strategic  tool  to  discipline  the 
thinking  process 

An  analytical  tool  to  structure 
an  analysis/  analyze  the 
organizational  structure  of  an 
analysis: 

Collaborative  tool  to  foster 
discussion 

A  collaboration  tool  to  provoke 
discussions  and  to  query  ideas  and 
assumptions; 

Foster  group  discussion 

Quality  control  (decompose  and 
show  logic  in  argument) 

Verify/Assess  completeness  of  the 
intelligence  analysis 

Validate  an  argument 

Collection  planning 

Prioritization  of  collection 
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Group  1 

Group  2 

Group  3 

management 

Teaching  tool 

Training  tool;  accelerate  learning 

Tool  to  argue  for  resource 
requirements 

Resource  forecasting 

Process  to  manage  analysts 

Explore  topic  in  depth;  good  to 
identify  relationship  between 
concepts  while  exploring  topic 

Structure  the  intelligence  problem 

Audit  and  defensibility  of  the 
analysis 

Management  of  information 

Table  4-1  demonstrates  the  following  four  potential  uses  that  all  three  groups  identified: 

•  Visualization:  CMapping  is  an  “imaginative  tool”  useful  for  providing  a  visual  approach  to 
thinking. 

•  Analytic  tool:  CMapping  can  be  considered  an  analytical  tool  that  models  the  process  of 
analysis  and  the  cognition  of  analysts.  This  capability  of  CMapping  can  be  useful  for 
disciplining  and  organizing  analytic  thought  and  structure  the  thinking  process. 

•  Collaboration  tool  to  facilitate  discussion:  CMapping  is  a  useful  collaborative  tool  that  can 
facilitate  discussions  among  analysts  and  to  query  ideas  and  assumptions. 

•  Validation:  CMapping  can  be  used  as  quality  control  to  validate  arguments  by  verifying  the 
completeness  of  an  analysis  and  identifying  any  gaps  in  the  argument. 

Despite  the  overlap  between  the  different  groups,  some  groups  identified  other  potential  uses  of 
CMaps  or  CMapping  in  general.  Groups  1  and  3,  for  instance,  identified  CMapping  as  a  way  to 
prioritize  collection  planning.  Groups  2  and  3  meanwhile  both  identified  CMapping  as  potentially 
useful  to  forecast  the  resources  needed  for  a  particular  intelligence  project.  Groups  1  and  2 
also  identified  that  the  representation  of  the  “map”  associated  with  a  topic  area  can  also  be 
useful  in  training  and  teaching  of  intelligence  analysis.  This  “product”  approach  to  considering 
the  CMap  was  given  little  attention  by  any  of  the  three  groups. 

As  a  product,  a  CMap  can  be  used  to  communicate  ideas  quickly  and  effectively.  For  instance, 
if  a  particular  person  of  interest  has  a  broad  web  of  associates  from  which  the  intelligence 
community  can  draw  clues,  a  CMap  is  an  efficient  means  of  showing  these  relationships  and 
identifying  who  of  these  associates  is  worthy  of  significant  allocation  of  resources.  As  a  product, 
the  CMap  can  also  provide  continuity  to  intelligence  files.  It  may  be  the  case  that  intelligence 
officers  are  rotated  to  new  responsibilities  as  their  knowledge  of  a  particular  area  reaches  a 
critical  mass.  This  critical  mass  may  be  the  point  where  they  are  able  to  extrapolate  from  the 
facts  to  the  likely  future.  A  detailed  CMap  for  a  particular  intelligence  area  of  interest  may  more 
effectively  transfer  the  sum  total  of  the  outgoing  intelligence  officer’s  knowledge  about  a  topic  to 
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the  incoming  intelligence  officer.  The  graphical  representation  makes  it  more  likely  that  the 
incoming  officer  will  use  the  information,  because  it  is  very  accessible  with  minimal  study. 

The  idea  of  continuity  also  extends  to  training.  By  having  acceptable  models  of  intelligence 
analysis,  like  the  model  (CMKMIA)  constructed  by  DRDC  Toronto,  intelligence  training  agencies 
can  foster  a  greater  degree  of  consistency  between  analysts.  These  models  can  serve  to 
represent  and  externalize  expert’s  mental  models.  While  consistency  may  not  be  desirable  from 
the  perspective  of  diversity  of  opinion,  it  is  desirable  to  have  all  analysts  following  the  same 
process  and  proceeding  from  the  same  basic  level  of  knowledge.  Training,  especially  of  skilled 
pursuits,  can  be  highly  variable  in  its  quality  due  to  the  teacher,  the  student,  and  the  learning 
opportunities  offered.  CMapping  can  help  to  standardise  some  of  this. 

In  the  case  of  training,  significant  effort  would  need  to  be  expended  at  the  outset,  in  order  to 
develop  a  comprehensive  model  and  then  for  it  to  be  reviewed  and  accepted  by  stakeholders. 
Teaching  of  intelligence  would  then  follow  the  curriculum  laid  out  by  the  CMap.  At  once,  the 
student  would  ‘see’  the  whole  of  the  expert’s  mental  model  of  intelligence  analysis,  but  they 
would  only  be  responsible  for  knowing  (or  learning)  a  manageable  chunk  of  that  model. 
Practically,  this  means  that  the  student  learns  the  content  being  presented,  but  is  not  fooled  by 
the  pedagogical  approach  into  creating  a  mental  model  that  will  need  to  be  deconstructed  and 
reformed  at  a  later  date  as  new  information,  relationships,  and  nuances  about  the  model  are 
added.  This  is  a  common  observation  in  learning  of  skilled  occupations:  opportunities  to  learn 
through  case  studies  or  simulation  are  degraded  as  students  first  ‘unlearn’  then  learn  again 
mental  models  that  are  sufficient  to  understand  the  situation  and  act  appropriately. 

There  is  a  further  advantage  of  providing  an  expert  mental  model  from  the  outset  of  training. 
Students’  retention  is  better  when  the  framework  being  developed  is  accurate.  A  CMap  is 
constructed  of  concepts,  related  by  linking  phrases.  This  representation  allows  a  reader  to 
quickly  identify  the  most  critical  concepts:  those  with  the  most  incoming  and  outgoing  links. 
These  concepts  form  the  backbone  of  a  training  curriculum.  With  this  framework  in  place,  new 
knowledge  is  more  easily  ‘located’  in  the  framework  and  has  a  higher  strength  of  relationship, 
leading  to  greater  probability  of  quick  and  accurate  recall,  long-term  retention,  and,  ultimately, 
job  success. 


4.2.2  Benefits  of  CMapping 

The  benefits  of  CMapping  that  were  identified  during  the  break-out  session  discussions  for  all 
three  groups  are  tabulated  in  Table  4-2  below.  In  common  with  the  list  of  potential  uses  shown 
in  the  previous  section,  there  is  much  overlap  between  the  groups  in  regards  to  the  benefits  a 
CMap  knowledge  model  or  CMapping  in  general  for  the  intelligence  community. 

Table  4-2:  List  of  Benefits  Related  to  CMapping  Derived  by  the  Groups 


Group  1 

Group  2 

Group  3 

It  allows  the  visualization  of 
relationships  between  concepts 

Allows  for  deeper  understanding  of 
problem  space  by  looking  at  all 

A  graphical  approach  to  links 
concepts 
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Group  1 

Group  2 

Group  3 

potential  relationships 

Good  to  foster  discussion 

Promotes  conversation  and  shared 
understanding 

Communicate  ideas 

Ability  to  link  propositions  in  an 
explicit  manner 

Explicit/Externalizes  process  and 
makes  the  process  transparent; 
objectifies  the  process 

Explicit  use  of  verbs  and  the 
ability  to  link  concepts 

Identify  gaps  and  anomalies  in 
the  argument. 

Identify  completeness  of  an  argument 

Allows  you  to  see  what  is 
missing/gaps  in  logic 

Good  for  visual  analyst 

Good  for  visual  people 

Can  see  the  bigger  picture 

Get  the  full  picture 

It  is  an  unconstrained  process  which 
can  be  considered  for  complex 
problems 

Capture  visually  very 
complex  thinking;  identify 
capabilities  within  a  system 
(i.e.  system  of  systems 
analysis)  and  allows  the 
breakdown  of  a  problem  into 
concepts  and  sub-concepts 

Makes  a  person  think  by 
creating  dissonance 

Gives  lots  of  options/answers 

Can  be  a  time  saver  (without 
wasting  time  writing  it  on  paper 
first) 

It  is  a  simple  process  (only 
some  participants  saw  this 
benefit) 

Allows  the  analyst  to  better 
match  the  analysis  to  the  end 
requirement 

The  graphical  approach  allows  a 
quick  assessment  of  the  whole 
problem  space 

The  graphical  approach  acts  as  a 
visual  checklist  of  concepts 

Ability  to  identify  resources  involved 
in  an  analysis. 

Can  be  adapted  for  the  recipient 

Higher  level  of  abstraction 

Externalized  thinking. 

Identify  range  of  probabilities 

Create  decision  tree;  expand 
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Group  1 

Group  2 

Group  3 

the  mind  of  the  decision 
makers. 

Good  for  conceptual  or 
systemic  questions 

Table  4-2  indicates  that  all  three  groups  identified  the  following  four  benefits: 

•  Visualization  of  relationships  between  concepts.  CMaps  enable  the  visualization  of 
relationships  between  concepts.  This  helps  analysts  to  gain  a  deeper  understanding  of  the 
problem  space  by  linking  concepts  and  looking  at  all  potential  relationships. 

•  Fosters  discussion/communication.  CMap  is  a  medium  in  which  analysts  can  communicate 
their  ideas  to  each  other  and  foster  discussion  about  particular  topics.  This  is  particularly 
useful  when  using  CMapping  as  a  collaboration  tool  and  to  ensure  a  shared  understanding 
about  a  topic  or  problem. 

•  Externalizes  the  problem  scope.  CMapping  objectifies  and  externalizes  the  analytical 
process.  CMaps  allow  an  analyst  to  link  propositions  in  an  explicit  manner  using  actionable 
verbs  which  can  be  useful  for  disciplining  the  thinking  process. 

•  Identify  gaps  and  anomalies.  CMapping  enables  an  analyst  to  systematically  identify  what 
is  missing  in  their  argument  or  gaps  in  their  logic.  Such  a  validation  tool  for  verifying  the 
completeness  of  an  analysis  and  identifying  any  gaps  in  the  argument  would  be  extremely 
beneficial. 

Despite  the  overlap  between  the  different  groups,  some  groups  identified  other  potential 
benefits  of  a  CMap  knowledge  model  or  CMapping  in  general.  Two  other  overlapping  benefits 
were  identified  by  Group  1  and  Group  3.  Both  noted  that  CMapping  may  benefit  people  who  are 
visual.  Both  groups  also  recognized  that  CMapping  would  be  beneficial  for  an  analyst  to  gain  a 
full  picture  of  the  problem  scope.  Finally,  Groups  2  and  3  both  recognized  that  CMaps  would  be 
beneficial  to  visually  capture  complex  problems  and  thinking. 


4.3  Implementation  issues  of  CMapping 

During  the  first  and  second  break-out  sessions,  several  drawbacks  and  difficulties  with 
CMapping  were  identified.  The  drawbacks  and  difficulties  that  were  derived  during  the  break¬ 
out  session  discussions  for  all  three  groups  are  tabulated  in  Table  4-3  below. 


Table  4-3:  List  of  Drawbacks  Related  to  CMapping  Derived  by  the  Groups 


Group  1 

Group  2 

Group  3 

CMapping  is  a  difficult, 
complex  process 

CMapping  is  very  difficult 

Difficult  process 

Not  good  for  linear  analysts 

Not  for  non-visual  thinkers 

Not  for  non-visual  thinkers 
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Group  1 

Group  2 

Group  3 

Yet  another  tool  (e.g.  i2) 

Added  value  unclear 

Other  similar  tools 

Difficult  to  decompose  a  vague 
idea 

Difficult  to  link  terms 

Difficult  to  define  terms  and  link 
propositions 

Cannot  weight  the  propositions 

No  weighting  factor 

Tool  needs  validation 

No  guarantee  of  better  understanding 

Not  self  explanatory/  Answer 
not  readily  clear 

Answer  is  not  readily  available 

Not  good  for  discrete  questions 

Can  answer  only  a  narrow 
range  of  questions.  Not 
intuitive  how  to  map  things  of 
“what  can  be”. 

May  be  seen  as  overly  ‘artsy’  in  the 
training  context 

Linear  method  may  be  more 
appropriate  for  highly  complex 
problems. 

Measure  relative  quality  of  outcome 

Need  good  focus  question 

Constrained  by  focus  question 

Time  consuming  /  strategic  and  not 
tactical  for  quick  turnaround 

The  CMap  needs  to  be  well 
crafted 

Can  have  a  lot  of 
overlap/redundancy 

Not  sure  how  to  use  active  and 
passive  voice 

Will  spend  energy  on  process 
rather  than  issue 

Not  practical  in  the  field 

Takes  time  to  learn 

It  is  not  a  presenting  tool 

Table  4-3  clearly  demonstrates  four  overlapping  drawbacks  that  were  identified  by  all  three 
groups.  Firstly,  it  was  quickly  determined  following  the  second  discussion  activity  that  all  three 
groups  found  the  process  of  CMapping  difficult.  Indeed,  as  was  mentioned  in  Section  4.1 .2.2 
the  process  of  decomposing  an  intelligence  problem  is  difficult  for  analysts  because  they  have 
internalised  the  skills  and  knowledge  required  performing  intelligence  analysis  and  thus  they 
have  difficulty  verbalising  this  information.  Secondly,  all  three  groups  also  recognized  that  while 
there  are  many  benefits  of  using  CMaps  and  executing  a  CMapping  process,  this  may  only 
benefit  people  that  are  visual  thinkers.  All  three  groups  recognized  that  the  added  value  of 
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CMapping  and  CmapTools  is  not  readily  clear  as  there  are  other  similar  tools  (e.g.  i2)  available 
for  the  intelligence  community.  Finally,  all  three  groups  had  difficulty  in  defining  terms  and  in 
linking  propositions. 

There  was  much  overlap  between  Groups  1  and  2.  Groups  1  and  2  both  identified  that  a 
drawback  of  CMapping  is  the  fact  that  there  is  no  guarantee  that  CMapping  will  provide  a 
greater  understanding  than  current  methods  being  used.  Indeed,  Group  1  recognized  that  the 
tool  still  needs  to  be  validated  as  it  is  still  in  the  experimental  stage.  These  groups  also  realized 
that  the  tool  lacks  the  ability  to  weight  a  concept  or  a  relationship  and  that  the  answer  is  not 
readily  clear  from  the  CMap. 

These  drawbacks  and  difficulties  could  impact  the  implementation  of  CMapping  for  the 
intelligence  community.  Below  are  the  potential  implementation  issues  related  to  the  potential 
uses  of  CMapping  for  the  intelligence  community? 

•  May  take  time  to  learn  the  process  and  the  CmapTools.  As  can  be  seen  in  Table  4-3  and  in 
Section  4.1 ,  all  three  groups  had  difficulty  constructing  a  CMap.  Indeed,  all  three  groups 
admitted  that  CMapping  is  a  difficult  process.  While  CMapping  is  not  a  complex  process  (in 
and  of  itself  notwithstanding  the  comments  of  Group  1  that  CMapping  was  quick  and 
intuitive),  some  assumptions  are  not  obvious  and  need  to  be  taught.  For  instance,  Group  2 
and  Group  3  were  unclear  how  to  phrase  the  linking  phrases  within  the  propositions.  There 
was  some  confusion  over  the  need  to  use  the  active  or  passive  voice.  In  addition,  Group  3 
did  not  realize  that  the  arrows  within  the  propositions  could  be  multi-directional  (i.e.  outgoing 
and  incoming).  Meanwhile,  while  not  explored  during  the  break-out  sessions,  CmapTools 
itself  would  require  some  basic  training  in  order  to  learn  how  to  use  the  tool  effectively. 
These  issues  imply  that  a  short  period  of  ‘formal’  instruction  would  be  needed  to  learn  the 
process  and  CmapTools  in  order  to  develop  effective  and  useful  CMaps.  The  training  time 
required  may  impact  the  intelligence  community’s  willingness  to  implement  CMapping  to 
perform  everyday  analytic  tasks.  A  separate  issue,  but  also  related  to  time,  was  that 
CMapping  was  seen  by  several  participants  as  not  amenable  to  quick  tactical  problem 
analysis  and  may  therefore  not  be  seen  as  a  practical  process  to  implement. 

•  CMapping  may  only  be  useful  for  visual  thinkers.  All  groups  agreed  that  CMapping  is  a 
useful  “imaginative  tool”  to  visualise  an  analyst’s  argument  or  concept.  It  has  been  noted 
however  that  it  may  not  be  useful  for  non-creative,  non-visual,  linear  thinkers.  Indeed,  it  was 
proposed  that  linear  analysts  may  get  lost  in  the  lines.  Therefore,  it  is  unclear  at  this  time  if 
CMapping  is  appropriate  for  everyone.  The  unconstrained,  visual  and  non-linear  process  of 
CMapping  may  not  be  readily  adopted  by  non-creative,  non-visual,  linear  thinkers  and  may 
pose  an  issue  for  implementation  within  the  intelligence  community.  However,  the  visual 
representation  may  be  particularly  well-suited  to  senior  decision  makers  who,  anecdotally, 
often  exhibit  a  preference  for  visual  representations  of  issues,  problems,  and  plans. 

•  CmapTools  does  not  have  the  ability  to  weight  the  propositions.  CmapTools  lacks  the  ability 
to  weight  a  concept  or  a  relationship  in  terms  of  its  overall  importance.  It  was  not  evident  to 
participants  how  conditionals  could  be  represented  in  the  map  because  the  links  between 
concepts  could  not  be  prioritized.  Participants  felt  there  was  a  need  for  a  tool  that  could 
blend  this  capability.  For  instance,  the  intelligence  analysis  application  i2  has  this  capability. 
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The  lack  of  the  weighting  capability  of  CmapTools  may  deter  some  people  from  the 
intelligence  community  from  adopting  CMapping. 

•  Unclear  how  CMapping  is  better  than  other  current  processes.  Some  participants  saw 
CmapTools  as  yet  another  tool.  Indeed,  the  intelligence  community  is  already  using  other 
similar  processes  with  tools  like  Mind  Manager  and  \2.  In  addition,  participants  could  not 
see  how  CMapping  could  guarantee  a  better  outcome  than  current  processes.  It  is 
conceivable  that  this  perception  was  partly  the  result  of  a  lack  of  training  in  CMapping. 
Perhaps  a  better  understanding  of  the  CMapping  process  and  CmapTools  may  impact  this 
perception. 

•  Developing  an  ‘accepted’  CMap  would  be  a  significant  undertaking.  Given  the  potential 
extent  of  such  a  model  for  intelligence  analysis,  the  process  by  which  such  a  model  would 
be  developed  would  necessarily  take  a  long  time  to  construct,  and  even  longer  to  validate 
with  stakeholders.  This  time  would  compromise  the  time  to  orient  an  audience  to  a  model, 
and  then  time  to  allow  the  audience  to  comment,  then  time  to  incorporate  (and  deconflict) 
the  audience’s  input.  To  this  end,  the  DRDC  Toronto  contribution  is  a  valuable  and  effective 
start.  Unfortunately,  during  this  workshop,  there  was  not  enough  time  to  validate  CMKMIA. 
Participants  were  encouraged  to  review  the  map  at  their  own  pace  and  provide  any 
feedback  directly  to  the  project  SA. 


4.4  Suggestions  for  improvements 

Several  suggestions  for  improvements  were  identified  during  the  workshop.  These  are  listed 

below: 

•  Implement  the  capability  within  CmapTools  to  weight  concepts:  As  was  mentioned  in  the 
previous  section,  CmapTools  does  not  have  the  ability  to  weight  the  propositions. 
Participants  suggested  that  there  is  a  need  to  implement  this  capability  within  CmapTools. 
This  ability  would  not  only  allow  propositions  to  be  weighed,  it  would  allow  an  analyst  to 
show  confidence  in  their  level  of  knowledge  and  identify  what  is  missing  from  their 
argument. 

•  Implement  the  ability  to  insert  3D  linkages  between  CMaps :  It  was  suggested  that  it  would 
be  useful  to  have  the  ability  to  create  linkages  between  maps.  This  ability  could  be  useful 
for  showing  relationships  between  CMaps  as  opposed  to  simply  having  a  sub-map.  It  was 
also  suggested  that  tracking  those  linkages  would  be  useful. 

•  Implement  basic  training  prior  to  introducing  CMapping.  As  was  mentioned  in  the  previous 
section,  participants  found  CMapping  a  difficult  process  that  takes  time  to  learn.  To  avoid 
confusion  and  erroneous  assumptions,  it  is  recommended  that  basic,  brief  training  sessions 
in  CMapping  and  CmapTools  be  given  prior  to  implementing  CMapping  to  the  intelligence 
community. 
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5  CONCLUSIONS 

The  discovery  workshop  brought  together  SMEs  from  the  intelligence  community  to  generate 
feedback  regarding  the  organization,  design  and  the  potential  application  of  CMapping  (as 
implemented  currently  in  CmapTools),  and  any  implementation  issues  and/or  suggestions  for 
improvements.  The  results  of  the  workshop  will  help  guide  the  potential  utility  of  CMapping  for 
the  intelligence  community,  as  well  as  analyst  training. 

Overall,  participants  were  positive  about  the  potential  use  of  CMapping  for  the  intelligence 
community.  The  discussion  activities  were  engaging  and  were  successful  in  eliciting  valuable 
participant  feedback.  Participants  could  readily  see  many  potential  uses  of  CMapping  in 
general  and  identified  some  benefits  and  drawbacks  in  terms  of  these  uses. 

Despite  the  limited  knowledge  and  experience  with  the  CMapping  approach,  it  is  interesting  to 
note  that  there  was  much  overlap  among  the  three  groups  in  recognizing  the  potential  uses  of 
CMaps  and  CMapping  for  the  intelligence  community  and  in  identifying  benefits  and  drawbacks. 
The  second  discussion  activity  was  a  powerful  demonstration  of  how  difficult  it  actually  is  to 
create  a  CMap.  It  was  evident  that  a  lack  of  knowledge  and  expertise  in  CMapping  influenced 
the  groups’  ability  in  constructing  a  CMap.  Despite  the  difficulties  encountered  by  the  groups 
during  the  second  discussion  activity,  by  the  end  of  the  activity  all  participants  saw  the  value  of 
CMapping  as  a  means  of  structuring  the  brainstorming  process. 

In  terms  of  the  potential  uses  of  CMaps  for  the  intelligence  community,  all  groups  recognized 
that  CMaps  could  be  used  for  visualising  the  thinking  process,  as  an  analytical  tool,  as  a 
collaboration  tool,  and  for  validating  arguments.  Workshop  participants  identified  several 
benefits  that  make  CMaps  particularly  useful  including  the  ability  to  visualize  relationships 
between  concepts,  the  ability  for  analysts  to  develop  a  deep  understanding  of  an  issue,  to  foster 
discussion  among  analysts  and  to  identify  gaps  and  anomalies.  While  there  are  many  benefits 
of  using  CMaps  and  executing  a  CMapping  process,  this  may  only  benefit  people  who  are  visual 
thinkers.  Meanwhile,  CMapping  was  found  to  be  a  difficult  process  that  takes  time  to  learn  and 
may  not  be  amenable  to  quick  tactical  problem  analysis.  Consequently,  the  added  value  of 
CMapping  is  not  readily  clear  as  there  are  other  similar  tools  (e.g.  i2)  already  available  in  the 
intelligence  community. 

CMapping  is  new  to  the  Canadian  intelligence  community  and  the  workshop  hosted  by  DRDC 
Toronto  was  one  of  the  first  introductions  of  CMaps  and  CmapTools  to  the  community.  The 
workshop  was  successful  in  identifying  potential  areas  of  application  and  areas  for  further 
development  and  investigation.  As  was  identified  during  the  workshop,  CMapping  is  still  at  an 
early  stage  in  its  introduction  to  the  Canadian  intelligence  community  and  needs  to  be  validated. 
Indeed,  a  needs  analysis  of  the  intelligence  analysis  task  needs  to  be  performed  and  matched 
to  the  capabilities  of  CMaps  and  CmapTools.  Comparative  evaluation  of  CmapTools  with  other 
tools  currently  used  in  the  community  may  help  identify  its  unique  added  value.  The  results  of 
the  workshop  will  guide  the  development  of  DRDC’s  research  program. 
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6  LIST  OF  ABBREVIATIONS 


CAE  PS 

CAE  Professional  Services 

CBSA 

Canadian  Border  Services  Agency 

CDI 

Chief  of  Defence  Intelligence 

CFSMI 

Canadian  Forces  School  of  Military  Intelligence 

CMap 

Concept  Map 

CMapping 

Concept  Mapping 

CMKMIA 

Concept  Map  Knowledge  Model  of  Intelligence  Analysis 

DRDC 

Defence  R&D  Canada 

IAS 

International  Assessment  Staff  -  Privy  Council  Office 

IACC 

Intelligence  Assessment  Coordination  Committee 

SME 

Subject  Matter  Expert 

TRIG 

Thinking,  Risk,  and  Intelligence  Group 
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ANNEX  A  CMAPS 


Figure  A-1 :  Group  1  ’s  CMap  of  analytic  integrity  drawn  during  discussion  activity  #2 


-  A-1  - 
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Figure  A-2:  Group  2’s  CMap  of  analytic  integrity  drawn  during  discussion  activity  #2 

(Top-Section) 


Figure  A-3:  Group  2’s  CMap  of  analytic  integrity  drawn  during  discussion  activity  #2 

(Mid-Section 


-  A-2  - 
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Figure  A-4:  Group  2’s  CMap  of  analytic  integrity  drawn  during  discussion  activity  #2 

(Lower-Section) 


Figure  A-5:  Group  3’s  CMap  of  analytic  integrity  drawn  during  discussion  activity  #2 
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